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This proposal suggests the kinds of information that 
could be developed to inform choices and answer questions related to 
the national education goals and their specific targets. An important 
approach to providing this kind of information is being undertaken by 
the National Assessment Governing Board, which has used panels of 
judges to set achievement levels for grades 4, 8, and 12 using a 
methodology often used to set passing scores on tests. In tandem with 
this effort, the following proposals are offered, not for 
accountability to the national goals, but to help reach them: (1) 
using panels of judges to set achievement levels; (2) defining goals 
as proportions of student reaching National Assessment of Educationi.1 
Progress anchor points*'; (3) identifying schools that exceed 
expectations; (4) learning what teachers expect; (5) choosing levels 
necessary for a high achievement curriculum; and (6) setting goals 
for literacy. Progress toward the goals for the year 2000 will 
require this kind of information to establish believable targets to 
which people can commit. Three tables provide information about 
national achievement, and one figure illustrates levels of 
mathematics proficiency. (SLD) 



* Reproductions supplied by EDRS are the best that can be made ^ 

from the original document. Vc 
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Preface 



In the course of our work at the Policy Information Center, we 
deal with information we need and don't have, as well as what we do 
have. So from time to time, we make a "policy information proposal." 
Last year, it was Skills Employers Need: Time to Measure Them . This 
year, we propose an examination of the information needed to set 
specific education achievement goals and targets, and interim goals 
and targets. 

Under the aegis of the National Education Goals Panel, an exten- 
sive system has been created to track progress toward the national 
goals. Yet it takes different kinds of information to help the education 
community decide on specific goals and targets. It is this piece of 
business that is our concern, and we hope at least to stimulate a 
discussion of these information needs. 

I am indebted to the following at ETS for reviews and other 
assistance: Richard Coley, Archie Lapointe, Steve Koffler, Garlie Fore- 
hand, Ann Jungeblut, and Irwin Kirsch. None, however, bear responsi- 
bility for the views contained herein. Carla Cooper provided desktop 
publishing services. Joanne Pfleiderer did the editing. 

Paul E. Baiton 
Director 
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Introduction & Summary 



Our country has set national educational goals for the year 2000. A 
National Education Goals Panel is in place and will i^sue "report cards" 
each September, The president has proposed an education strategy, 
America 2000, and legislation supporting it has been introduced in the 
Congress. 

These national education goals are, of necessity, very general. 
Throughout the nation, educators, policymakers, and the public will be 
struggling, if all goes as intended, with translating general goals into 
specific targets. 

How much do we want Americans to learn in the education system? 
How literate do they need to be? What are realistic targets, as well as ideal 
ones, we should move toward? How can poliqanakers communicate why 
they think the chosen targets are desirable ones? How much literacy is 
needed for different kinds of occupations? 

This document suggests the kinds of information that could be 
developed to inform these choices and answer these questions. This is a 
"policy information proposal," not a synthesis of existing information. 

One important approach to providing information of this kind is 
being undertaken by the National Assessment Governing Board (flAGB), 
the policy-making body of the National Assessment of Educational 
Progress (NAEP). NAGB has used panels of judges to set "achievement 
levels" for grades 4, 8, and 12, using a methodology often employed to set 
passing scores on tests. These are being issued first in mathematics, based 
on the 1990 NAEP mathematics assessment. It will be some time before 
they will be available for all subjects. 

The extent to which these levels, or standards, will be used through- 
out the nation is yet to be seen. The NAGB effort represents one serious 
approach to providing information that can be used in establishing spe- 
cific targets for achievement. But there are other approaches to informing 
the choices that have to be made, and the better and more varied the 
information, the sounder will be the decisions. To the extent that the 
information is convincing as to what students need to know, and what is 
feasible because it is actually being accomplished somewhere in the U.S., 
the greater will be the publics commitment to attaining the goals. While 
Missouri calls itself the ''show me" state, this nickname applies to much of 
America as well. 

In tandem with the NAGB achievement, levels, the elements of this 
proposal, if implemented, would result in tlie following: 

♦ The ability to set targets now for all subjeicts NAEP 
assesses at the 4th. 8th, and 12th grades, in temns of the 
percent of students at or above certain levels on the 
NAEP scale. 
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• Average scores for a set of schools demonsirating much 

higher achievement levels than what would be expected, 
based on the prevailing relationship in the United States 
between socioeconomic status and achievement. These 
would be lead schools that are proving that we can raise 
student performance now . 

• The expectations of teachers of students who are 
assessed by the National Assessment of Educational 
Progress. These expectations would be elicited by asking 
a sample to identify the questions on the NAEP test that 
they would expect their students to answer correctly, 
based on what they have taught them. We would know 
the level of teachers' standards, the gap between them 
and students' achievement, and how teachers' 
expectations are responding to goals set for the year 
2000. 

• How new high achievement curricula (for example, 
curricula based on the new standards of the National 
Council of Teachers of Mathematics) translate into scale 
scores in NAEP, so we know where students stand in 
relation to these standards. 

• The levels on the literacy scales required for 

- broad bands of occupations 

- specific, benchmark, entry-level jobs 

- such things as successful accomplishment of buying a 
week's groceries, purchasing insurance, or operating 
new high-tech appliances 

The National Education Goals Panel is now engaged in a far-ranging 
exercise to determine the data the nation needs to track progress toward 
the established goals. But nothing comparable has been undertaken to 
create information to help establish specific targets and objeaives as the 
decade progresses. This modest set of proposals is offered in this spirit, 
not for accountability to the goals, but for assistance in reaching them. 

!• Panels of Judges to Set Achievement Levels 

There have long been a number of methodologies for setting "cut 
points" on tests to establish passing scores. These have been used in a 
variety of ways to establish standards for achievement. In one 
way or another, they require the exercise of judgment, although they 
usually are accompanied by sophisticated statistical and psychometric 
analysis. Other approaches described in this document are designed to 
produce information that policymakers and educators take into 
account in making their judgments about achievement goals; the NAGB 
approach produces goals for them. 
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A heroic effort is under way by NAGB, using the NAEP assessments 
and responding to a congressional authorization in 1988, to identify 
''appropriate achievement goals" in subjects NAEP assesses. Congress did 
not specify approach, and there was little, if any, legislative history on 
what it had in mind. 

The initial effort is to establish achievement levels in mathematics in 
grades 4, 8, and 12. In June 1990, NAGB appointed a panel of 63 judges, 
of whom about 70 percent were educators (representing mathematics 
teachers, college math instructors, principals, and state and district curricu- 
lum specialists). The remaining 30 percent were employers, civic group 
representatives, and interested citizens. The process has been extensive 
and included public hearings on the recommendations made by the panel 
of judges. 

Gregory Anrig, president of Educational Testing Service, testified at a 
hearing conducted by NAGB on January 8, 1991. He began by saying that 
"I congratulate the National Assessment Governing Board for vigorously 
pursuing this complex task and for your efforts to involve appropriate 
constituencies in the process." He then spelled out a complete methodol- 
ogy he thought was necessary to establish levels that would stand up to 
criticism, advising NAGB to resist issuing levels until all steps of an ac- 
cepted process had been completed. He established four major tasks to be 
accomplished: 

1. Developing the competency statements 

2. Obtaining empirical information to inform the 

standards 

3. Setting the standards 

4. Evaluating the process 

Once NAGB announces the achievement levels, or standards, NAEP 
assessment results would be reported in terms of the percent of students 
reaching the standards at grades 4, 8, and 12. The contemplated levels are 
"basic," ••proficient," and "advanced." An example of the proficient level 
at grade 8 follows:^ 

Students at the proficient level should be able, with and without 
a calculator, to solve problems using fractions, decimals, 
percents, rates, proportions, similar figures, algebraic formulas 
and functions, and understand and use exponents. These 
leamers should recognize elementary algebraic properties and be 
able to use algebi*aic procedures to solve equations and 
inequalities. For example, proficient leamers should be able to 
solve problems involving measurement, understand and apply 
measures of central tendency and basic sampling techniques, and 
know the properties of plane and solid geometric figures. These 
students should be able to apply logical problem-solving 
strategies, such as working backward to a solution, making tables 



Quotui3 from (icAcripuom avaJiblc from NAGB in January 1991 
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and i;?raphs, and extending more complicated patterns. It is 
appropriate that they should know when and how to use a 
calculator. 

2. Goals as Proportions of Students 
Reaching NAIP '^Anchor Points" 

NAGB has chosen to use a judgment process to define what students 
should know and be able to do, as described in the previot^s section. 
Gregory Anrig, in his testimony of January 8, pointed our. that the NAEP 
assessment has been designed to measure what students now know and 
can do. This introduces constraints, at least until the assessment is 
changed. 

A simpler approach, based on what NAEP now does, is to use 
the "anchor points" established by NAEP beginning with the 1984 assess- 
ment. The judgment would involve setting goals for the percent of stu- 
denLs who should reach these anchor points in a specified period of time. 

While the NAGB is defining what students should know, the anchor 
points are a way of describing what they do know. NAEP identified tasks 
students (with a high degree of probability) can successfully perform, at 
50-point intervals on the NAEP scale. Then, it had math educators look at 
these tasks at each level and generalize about what they had in common. 
For example, at level 300 in mathematics (on a scale of 0 to 500), the 
description is as follows; 

Level 300 - Mndprat^ly Com plex Procedures and Reasoning. 
Learners at this level are developing an understanding of number 
systems. They can compute with decimals, simple fractions, and 
commonly encountered percents. They can identify geometric 
figures, measure lengths and angles, and calculate areas of 
recungles. These students are also able to interpret simple 
inequalities, evaluate formulas, and solve simple linear equations. 
They ciin find averages, make decisions on information drawn 
from graphs, and use logical reasoning to solve problems. They 
are developing the skills to operate with signed numbers, 
exponents, and square roots. 

For example, we know that the percent of 13-year-olds who reached 
this level was just l6 percent in 1986, compared to 18 percent in 1982 and 
1976.^ We could set goals in terms of agreement on a target for 1994, 
1998, and so forth, such as 25 percent, or 33 percent, or 50 percent. The 
same could be done for Level 250 (Basic Operations and Beginning 
Problem Solving) and Level 350 (Multi-Step Problem Solving and Algebra). 

This method could be used in the interim until levels are established 
by NAGB in all subjects at 4th, 8th, and 12th grade levels. (It has been 
three years since Congress authorized NAGB to set achievement goals, and 
they have just been completed in mathematics. A request for proposals 



^However level 300 was rc^cTtncd by NA£P in 1990, k> it would be Dcc«5Mry lo iurt with 1990- 
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has just been issued to comn^'-nce work in otiner subject areas.) Or, some 
may continue to be more cc iortable with the empirically established 
anchor points than the judgmentaliy established achievement levels, 

In November 1989, the Policy Information Center published Informa- 
lion For National Pcrfo rmanrf r,nn|,^ For V^\\r?Uon: a Workhnnk This 
workbook, which utilized the approach described above, stated that 

" While we have provided average proficiency scores (meam), we 
have emphasized diMnhliUom for setting goals. This way, the 
wide variations among students of the same age and grade are 
recognized. The NAIIP scales, by describing what 
students know and can do at each level on the scale, aids 
judgments about the percent of students who should attain a 
particular level. We can deal with questions such as: Do we 
propose to raise more students at the bottom toward the m'^dle? 
Raise studettts in the middle toward the top!'" 

Subsequently, when the national goals were issued in February 1991, 
they called For "raising achievement in every quartile." The "workbook" 
approach Facilitates setting goals in those terns, and NAEP anchor points 
are now available For reading, writing, mathematics, science, history, 
civics, and geography. 

That Exceed Expectations 

Education goals can seem abstract when thr y are created through 
some complicated technical procedure; in Faa, they can seem so far From 
reality that they strain credibility. One way to establish believable goals is 
to show thai there are schools already attaining them, right here m the 
early 1990s. We can be reasonably sure that such schools exist, but it is 
not so apparent that they can be easily identified. 

It is hard to identify these exemplars because there is large variation 
in Factors and conditions related to educational achievement in the U.S. 
The socioeconomic characteristics oF student bodies vary. The resources 
available to schools vary. The majority-minority makeup oF student bodies 
varies. 

Statistical analyses can deal with these variations in terms oF estab- 
lishing their relationship to student achievement. The association between 
these factors and achievement can be quantified. Given what we know 
about a student body and a school, student achievement can be prediaed 
with differing degrees of accuracy, and the "expected" level of achieve- 
ment can be compared with the "acti il" level. This approach has been 
used as far back as 1962 in Japan to ,..^e in which geographical areas the 
schools were doing better than, or worse than, expected. It is now used 
by the state of California. 

While strong statistical associations will be found, there will be 
schools in which student performance exceeds what these associations 
predict (as well, of course, as schools performing less well than pre- 



dicied). A statistical analysis using a large body of data, such as NAEF\ 
could be used to establish the association and identify a ^ of schools 
exceeding cxpectations-noi individual schools. The aQhievement levels oi 
thc ^'se scho ols repr esent levels that are anainable . Knowing what these 
levels are is information that could be used to establish realistic goals for 
education achievement, or at least interim goals toward more ambitious 
ones. In statistical terms, these schools would be called "outliers" in 
regression analysis.^ 

These school achievement levds could be clustered at different 
points along the continuum of student and school conditions. The 
average scores for a set of schools that considerably exceeded expeaed 
scores couid De computed at each quartile of student and school condi- 
tions, say, the top 10 schools (of course, they would not be identified by 
name). A table could be produced such as the one below: 



Table 1 





Math Scores (Median) for Eighth-Grade Students 


Student and 
School Conditions 


Aanal Scores 


Combined Score, Top Ten 
Schools Exceeding Expectation 


1st Quartile 






2nd Quartile 






3rd Quartile 






4th Quartile 







The experiences with outlier studies in the 1970s should be dra^vn 
upon in doing the kind of analysis described above. Large samples are 
necessary. Averaging scores from at least 10 schools will produce more 
reliable results. 

4. What Teachers Expect 

At the classroom level, where education takes place, achievement 
goals are set by the individual teacher. We know how these students score 
on NAEP. But how do students score relative to how their teachers expect 
them to score? And how do expectations of teachers compare with 
achievement goals set by others, including the National Education Goals 
Panel? And how are these teacher expeaations changing? 



*ln the 19705, there was a lot of mtcrcsl in "ootliefs." often sinking ooi tndividujJ schools These studic* f»n into K>me problems, p»rtic\ilirlv becatuc individuil outlier schools 
were not very «tablc. Put experiences, as veil ss advinccs m sutitucil methods, will r>c<d to be drawn on for fuove analyses 
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We can find out by asking a sample of teachers to identify the 
questions on the test that they would expect most of their students to 
answer correaly, based on what they have taught them and what they 
know about their students' prior preparation/ This would not be an 
attempt to match the scores of students in a class with the expectations of 
their particular teachers (teacher scores would be confidential, and indi- 
vidual teachers would not be identified). Instead, the average of teacher- 
expected scores for a subject at a grade level could be compared with that 
of the students, and teacher expeaations tracked over a period of time. 

We need to have goals in terms of what teachers expect, as well as 
goals what students know and can do. Raising these expectations, based 
solidly on what is taught, will be a key to raising achievement and reach- 
ing national goals. Using the test itself is a direa way to identify these 
expectations; it would enable completing the blanks in Table 2, Such a 
system would permit going beyond national averages, to geographic 
differences or subpopulations. 

These expectations are teachers' immediate goals; their goals, and 
how they change, will be leading indicators of achievement changes. Of 
course, when a teacher analyzes a NAEP test for this purpose, it will not 
be a perfect measure of what a student has been taught (and of course, 
how well he or she has been taught); as a praaical matter, no such per- 
fea measure is possible. But it is about as close as we are likely to get. 







Table2 




Teacher Expected Scores 




4ih Grade 


8lhG»de 








Recoil 


State 






State 


Natkm 


Region 


Stale 


Reading 




















Writing 




















Madienrtatics 




















Soenoe 




















Histoy 




















Civics 




















Geography 





















^$ ts in ovcrwmplicition In pticucc, « methodology would likdy be med in which the tc*chcr» rate the items in lerms of t percent e3q>ccted lo get chem cof recL Bm 
tundard methodologies do exist 
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5. Levels Necessary^ for a High 
Achievement Curriculum 



One question to ask when choosing a level for achievement goals is: 
What level on a NAEP scale is associated with mastery of a specific text or 
curriculum that is judged to be a good one? In a sense, a procedure would 
be followed in which the text (or curriculum) would "take" the NAEP 
assessment. That is, if an individual mastered what was in the text or 
curriculum, what score would he or she get on a NAEP scale? 

A related question is where does the person need to be on a NAEP 
scale at the end of one academic year to begin a text/curriculum at the 
beginning of another? 

Where would that person be on the NAEP scale after successfully 
completing, in the 12th grade, the curriculum called for by the National 
Council of Teachers of Mathematics? 

At the "top" level of achievement, where is a passing grade (3) on an 
Advanced Placement examination in terms of a NAEP scale in that sub- 
ject?^ 

These are the kinds of questions that are likely to be in the minds of 
a panel of judges asked to establish a cutscore in setting standards. To a 
considerable degree, however, these are empirical or analytical questions 
that can be answered with more than the approximations of a panel of 
experts, with widely different backgrounds, examining assessment items 
and making judgments. (And, if such panels had such information, they 
would be able to make better judgments.) 

The figure on the next page illustrates the kind of information that 
could be developed, with varying degrees of precision. 

6. Setting Goals for Literacy 

One of the national goals set by the governors and the president is 
that all American adults be literate by the year 2000. Employers are con- 
cerned about insufficient literacy. A national movement has been under 
way for nearly a decade to wipe out illiteracy. 

But what constitutes literacy? What would be a level all adults should 
reach by the year 2000? Don't different life tasks have different literacy 
requircLients? What do we need to know to be able to set useful and 
realistic literacy goals? 

In the measurement of literacy, a consensus is emerging around the 
concepts and methods used in the Young Adult Literacy Study conducted 
by Educational Testing Service in 1985 (reported in 1986) for NAEP. ETS 
will undertake a similar study for all adults, in 1992. It will be called the 
National Adult Literacy Survey (NALS).^ 

» Aiwwcfing this would probably require giving the NAEP »»ae»3meni to a iimpJe of itudcnti who had taken the AP cxiimrutioo 

* This will be carried out under conraa ^tfh Nau jhaJ Center for Education Smtiaaca. _ 
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Figure 1 

Levels of Mathematics Proficiency 
(National Assessment of Educational Progress) 

Scale (0-500) 
500 



Multi-Step Problem 
Solving and Algebra 



Moderately Complex 
Procedures and 
Reasoning 



Basic Operations and 
Beginning Problem 
Solving 





Beginning Skills 
and Understanding 



Simple Arithmetic 
Facts 



m 



450 



400 



350 



300 - 



250 



200 -- 



150 - 



100 -- 



50 



Advanced Placement Students 



NCTM Standards Through Twelfth Grade 



TwelWi Graders In 1990 (295) 

Highest Ranking Stato in Eighth Grade (281) 

Eighth Graders in 1990 (265) 

Mastery of a ""Good' Fourth Grade Text/Cum'culum 

Fourth Graders in 1990 (216) 
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These studies identify literacy in three relatively distina areas^: 



• Prose literacy. Reading and interpreting prose such as that 
found in newspaper articles, magazines, and books. 

• Document literacy. Identifying and using information 
located in documents such as forms, tables, charts, and 
indexes. 

• Quantitative literacy. Applying numerical operations to 
information contained in printed material such as a 
menu, a checkbook, or an advertisement. 

The three types of literacy are measured on a scale of 0 to 500. Of the 
three, research has identified document literacy as being the kind most 
encountered in workplaces. In Figure 2, the results of the 1985 study are 
reported for young adults (21-25) who graduated from high school but did 
not go on to college. 

The kinds of tasks these young people are likely to be able to do are 
described at different points along the scale. 

As can be seen in Figure 2, 97 percent are at the 200 level or higher, 
where the representative task is being able to enter personal information 
on a job application, and praaically all reached the level of entering a 
caller's number on a phone message form. But at higher levels on the 
scale, represented by tasks that are moderately complex, the ranks thin 
out quickly: Only half reached the 300 level, represented by a task requir- 
ing following direaions from one location to another using a map. Just 11 
percent reached the 350 level, characterized by a task using a bus sched- 
ule to select the appropriate bus for given departures and arrivals. A 
similar picture emerges on the other two literacy scales. 

Few are illiterate, in the sense of being able to do none of the tasks. 
Literacy is a continuum, and no single point on the scale neatly and 
simply separates the literate from the illiterate. But die question must be 
asked: Literate for what function, what activity, what task, and what job? 

In the following section, we suggest several kinds of information that 
would help policymakers and others who must set standards in making 
informed judgments. 

A« Occupations and Literacy 

One way to look at literacy requirements is to look at the actual level 
of literacy of people who are performing particular activities. For example, 
what are the literacy levels of people who are now employed in particular 
occupations? This approach can be illustrated, in a rough way, by using 
the data from the 1985 Young Adult Literacy Study. In that study, the 
occupations of employed young adults were identified. 



'See Wo^k plare Co<np^t^« MeMUfin^^ L.tgfacv >nd Employment ReidmcM . by Piul E. Biiion »nd Irwm Kirtch. Govemfflcm Prinung Office. 1990. 
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Figure 2 — Percentage of high school graduates scoring at or 

above selected levels of proficiency in prose literacy, 
with tasks illustrative of various levels: 1986 



Proficiency 
scale 


Percent at 
or above 
selected 
levels 


Example tasks at various 
proficiency levels 


500 t 
400" 

375" 3.2 (0.8) 
350-- 12.2 (1.3) 
325-- 26.6 (1.8) 
300 -- 48.4 (l.-T) 
275" 66.4(1.4) 
250" 81.4 (1.3) 
225" 91.2 (0.9) 
200" 96.7 (0.6) 


371— Orally interpret distinctions between 
types of employee benefits 

340 — State in writing argument made in 
lengthy newspaper column 

313 — Locate information in a lengthy news 
article 

277 -Write a letter to state that an error has 
been made in billing 

262 — Locate information in sports article 
199 — Write about a job one would like 



NOTE: High School graduates include those who had some postsecondary experience, 
but no degree. Standard errors are shown in parentheses. 

SOURCE: National Assessment of Educational Progress. Uteracy: Profiles of America's 
Young Adults, 1986. 



ERLC 



In Table 3, we have reproduced a table published in 1990 in Work- 
place Competenrie.<;: Measuring Liter ary and Employment Readiness. 
Using the document literacy scale, the table shows the average document 
proficiency level for young adults employed in each of nine occupations 
who reported working full-time for a full year. For example, the average 
scale score for laborers was 277 (the lowest), and the average scale score 
for sales workers was 297. Table 2, shows what these scores mean in 
terms of representative tasks. Table 3, shows how various subgroups are 
arrayed along the scale. 

Young adult laborers are at about the 275 level, where they can do 
things such as use an index from an almanac. Seventy-three percent of all 
young adults (not just those employed full-time) are at or above this level, 
but just 40 percent of high school dropouts score this high or higher. 

A minimum goal of 275 would be approximately the average level of 
young employed laborers. Is that a minimum we should strive for, mean- 
ing that we need to get about one-fourth of the young adult population 
up to it? Or should we set a more modest goal of 250, with l6 percent of 
all young adults below that level (and larger proportions of Black and 
Hispanic young adults)? In Workplace Competencies , similar information 
is presented for the prose and quantitative literacy scales. 

As we said, these data are very rough. The nine occupations are 
very wide bands and often include a large number of specific occupations, 
with widely varying literacy requirements. For the young adults employed 
in each of these occupations, there is a distribution of scores; here we 
have used only the average. These data are not very useful to set goals for 
preparing for specific occupations, but only to inform those setting broad 
goals for a nation, a state, or a community. The information would be 
improved if the samples were larger, permitting more detailed breakouts 
of occupations an J a look at the distribution itself, selecting a narrower 
range of scores and excluding the extremes. And the data are for people 
who hold those jobs, not necessarily for the skill levels employers seek. 
But it is an approach that could be improved. 

An expansion of such literacy assessment information will be avail- 
able this summer, after the Workplace Literacy Project is completed by 
ETS, under contract with the U.S. Department of Labor. It will provide a 
literacy profile of three populations-applicants at the Public Employment 
Service, unemployment insurance claimants, and enrollees in Job Training 
Partnership Act QT?A) programs. Information will be available about jobs 
previously held. The project will also enable looking at the literacy levels 
of enrollees in different kinds of JTPA programs, suggesting an approach 
to examining the literacy levels necessary for such training programs. Also, 
the National Adult Literacy Survey (NALS) will be carried out in 1992, 
providing relevant information on all adults. 
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B. Literacy Audits 

The approach just described relies on information from large-scale 
assessments of adult literacy, linking the assessments with information 
bout the occupations of people assessed. A much more precise, and 
more costly, approach is to audil the literacy requirements of work in a 
specific job. 

In 1990 an ETS team completed the Job Literacy Project, developing 
and testing a new methodology called Job Literacy Analysis, a modification 
of the job analysis procedures used successfully by ETS over the last 15 
years. The project sets out to describe the literacy requirements of five 
entry-level jobs: 

• Food service worker 

• Nurse assistant 

• Secretary 

• Word processor 

• Data processing equipment repairer 

For each job, the projea produced a *'job literacy desciiption." 

The specific documents and materials used in each entry-level job 
were gathered and analyzed. In the ca.se of food-service workers, these 
included sandwich lists, food scales, charts, schedules, memos or notices, 
operational instruaions, reservation forms, diagrams, order sheets, menus, 
recipes, labels, food production orders, and safety sheets. The specific 
task^ these workers must perform were established (for example, ''mark 
price code or label on wrapped sandwiches"). 

There are a number of possible uses of such literacy audits, including 
creating tests and developing training programs. Through additional 
analysis, at some cost, the audit material can be used to locate the job on 
the three literacy scales previously described. This use is relevant here. 
Knowledge of literacy requirements for a set of jobs spanning the occupa- 
tional distribution would provide objective information for setting realistic 
goals for literacy. These goals would be tied to the literacy scales being 
used in large-scale assessments, that produce literacy profiles for the 
nation, and incieasingly, for individual states as well. 

Such benchmarks can be created now, with relatively small expense, 
using the literacy audits described above, and benchmarks for higher level 
occupations ca n be established with additional audits. 

Such info)*mation could be used to establish literacy targets for 
specific indiviciuals preparing to enter particular occupations. However, 
we are interested in their use for establishing policy goals for whole 
educational systems, states, and the nation, as well as goals for JTPA 
programs, JOBS programs under the Welfare Reform Aa, and "second- 
chance" institutions for high school dropouts. 



C. Literacy for Living 



Setting literacy goals is important for successful employment prepara- 
tion, yet this is only one area in which literacy is important for negotiating 
life. The literacy tasks used in the 1985 large-scale literacy assessments 
(and to be used in 1992) span most activities adults engage in. We talk 
about how the "information age" and the higher technological literacy 
required for using consumer products has raised literacy requirements. But 
we could establish this factually, and track change by applying the audit 
approach previously described. 

For example, what levels of prose, document, and quantitative 
literacy are required to intelligently perform: 

- A week's grocery shopping? 

- Understanding food coupons? 

- Reading sales ads in the paper? 

- Figuring out which size of a particular product hr.> the 
lowest unit cost? 

The tasks involved in a week's grocery shoppincj can be identified, 
and accomplishing them can be located on the literacy scales. The same 
can be done purchasing car insurance, or understanding the health care 
system, or using VCRs. 

With modest effort, we can become more literate about the literacy 
requirements of our society. Despite the large amount of public discussion 
about literacy in the last few years, we know very little about how much 
is required. 



♦ ♦ ♦ 



The information system to help educators, policymakers, and the public generally to 
establish specific education goals and targets is now very primitive, even as our statistical 
information is increasing. Progress toward the goals for the year 2000 will require better 
information, of the kinds sketched out in this paper, information for establishing believable 
targets-and interim targets-that a pragmatic people can become committed to. 
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